Operating device and method for detecting a user selection of at least one operating functon of the operating device

ABSTRACT

One, some or all of a specified number of multiple operating functions of an operating device can be selected by a user. Each operating function is assigned to a respective finger of a hand of the user in the operating device by a controller, and a detection device is used to detect a contact of at least one of the fingers of the hand on a specified contact surface of the operating device and ascertain which of the fingers of the hand is contacting the contact surface. Each of the operating functions whose assigned finger contacts the contact surface is determined as the user selection by a controller, and the user selection is signaled to a subsequent process of the operating device using selection data which identifies each selected operating function.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is the U.S. national stage of International ApplicationNo. PCT/EP2017/050915, filed Jan. 18, 2017 and claims the benefitthereof. The International Application claims the benefit of GermanApplication No. 10 2016 003 072.0 filed on Mar. 12, 2016, bothapplications are incorporated by reference herein in their entirety.

BACKGROUND

Described below is a method for detecting a user selection of one ormore or all of the operating functions from a predetermined set of aplurality of operating functions of an operating device. The userselection can be executed by the user using his fingers. Also describedare an operating device by which a plurality of operating functions aremade available, from which a user can make a user selection and a motorvehicle having such an operating device.

In a motor vehicle a central operating device can be made availablewhich permits a user of the motor vehicle to operate a plurality ofdifferent vehicle components by a respective operating function. Forexample, an operating function for an air conditioning device and anoperating function for a media playback device, for example a radio, andan operating function for the navigation device of the motor vehicle canbe made available. Each operating function can include, for example, auser interface, in particular a graphical user interface (GUI), for therespective vehicle component. If a user of the motor vehicle then wishesto operate a plurality of the vehicle components, the respectiveoperating function for the vehicle component must activate in theoperating device. This is generally possible only successively, whichmakes the operating process time-consuming and therefore laborious inthat it is necessary to change between the operating functions. If adriver wishes, for example, alternately to operate the radio, adjust theair conditioning device and make settings in the navigation device he isrequired to activate the corresponding operating function in theoperating device each time for this purpose.

An operating device for a motor vehicle can have, for example, atouchscreen for detecting user inputs. With respect to the detection ofuser inputs on a touchscreen, DE 10 2010 048 745 A1 has disclosed that aplurality of graphical objects can be displayed on the display face of atouchscreen, which objects can be selected or activated by touch. If, incontrast, a 2-finger operating process is detected on the touchscreen, asliding mode for the objects is activated in order to be able tore-arrange the objects on the display face.

DE 10 2008 022 752 A1 has also disclosed detecting a plurality ofcontact positions on a touchscreen and as a result controllingoperations during the operating process, for example the zooming of maprepresentations, the rotation of map representations or the selection ofan object.

DE 10 2008 048 825 A1 discloses a screen which has a gesture detectionunit by which an activation gesture is detected which detects theapproaching of a spread-out hand to the screen and subsequentlyactivates a modifying mode for the operating device.

Operating devices by which a plurality of operating functions are madeavailable are, however, not known in conjunction with motor vehicles.For example, a portable, mobile terminal, such as for example asmartphone or a tablet PC, also makes available a plurality of operatingfunctions, in each case one for operating a program or an application ofthe terminal.

SUMMARY

The method performs configuring in a more efficient way, in the case ofan operating device which makes available a predetermined set ofselectable operating functions, the use of a plurality of theseoperating functions.

The method detects a user selection of one or more or of all theoperating functions from a predetermined set of a plurality of operatingfunctions of an operating device. The operating device can be provided,for example, for operating a plurality of vehicle components for a motorvehicle or a plurality of applications of a mobile terminal such as, forexample, a smartphone or tablet PC. In the operating device, in eachcase a finger of a hand of a user is assigned to each of the operatingfunctions of the set by a control device. An instance of finger contactof at least one of the fingers of the hand, in particular a plurality offingers, is detected by a detection device on a predetermined contactface of the operating device. If a plurality of instances of fingercontact are detected, there is provision that it is a case ofsimultaneous instances of finger contact. Furthermore, the detectiondevice determines which of the fingers of the hand contacts the contactface, that is to say whether, for example, the thumb and an index fingeror the thumb and a middle finger carry out the respective one instanceof finger contact. The contact face can be, for example, a surface, forexample the surface of a touchscreen or the surface of a touch-sensitiveoperating face, that is to say what is referred to as a touchpad, orelse a virtual contact face which can be freely defined in space. Eachof the operating functions whose assigned finger contacts the contactface is specified as the user selection as a whole by the controldevice. The user selection is signaled to a subsequent process of theoperating device by selection data which identify each selectedoperating function. The selection data therefore makes availableinformation as to which of the operating functions the user has selectedusing his fingers.

The method has an advantage in that a user can select a plurality ofoperating functions simultaneously and in this context the selection canbe any desired combination of the selectable operating functions of theset which are present, since the user can specify individually with eachfinger which of the operating functions he would like to select. In thiscontext, there is in particular, of course, provision that in each caseanother of the operating functions is assigned to each finger.

According to one development, the user selection is only specified if aconfirmation gesture of the hand is detected by the detection device.The user can therefore execute this instance of finger contact on thecontact face with at least one finger and can provisionally mark anoperating function without as a result initially specifying the userselection. Only after he has also executed the confirmation gesture isthe user selection specified. This provides the advantage that the usercan still move his fingers around on the contact face, that is to saycan change the finger contact, and only confirms the conclusivecombination of instances of finger contact by the confirmation gesture,and as a result specifies the user selection.

In order to avoid inadvertently changing the finger contact combinationwhen the confirmation gesture is executed, one development provides thatan overall movement of the hand together with the fingers which contactthe contact face in a predetermined direction along the contact face isdetected as the confirmation gesture. The user can therefore retain theposition of the fingers on his hand and merely has to move his entirehand in the predetermined direction. This prevents the combination ofthe instances of finger contact being changed when the confirmationgesture is executed.

In order to make the selection of the operating functions more easilyrecognizable, a development is provided in which the operating functionsof the set, that is to say the selectable operating functions, arerespectively represented by a dedicated graphical object on a displayface, that is to say, for example, displayed on a screen. Such agraphical object can be, for example, an icon or lettering. Therespective graphical object of each operating function, excluded fromthe user selection, that is to say each non-selected operating functionof the set, is deleted from the display face. Each object which isselected by the user selection continues to be displayed. As a result,the user can use a single selection to remove all those graphicalobjects which he does not wish to have displayed. The combination of theobjects to be deleted, or the remaining objects, can be set as desiredhere, since one finger can be used for each graphical object. By thisdevelopment, a user can adapt, for example, a menu structure to hisrequirements, that is to say can individualize the menu structure. Forthis purpose, he uses finger contact to retain as graphical objectsthose menu entries which he would like to use.

The graphical objects may be displayed in a different spatial regionthan the one in which the finger contact of the at least one finger isdetected. The user therefore does not have to use his fingers to contactthe respective graphical object but instead can execute the fingercontact on a contact face next to the graphical objects. As a result,when the fingers are applied to the contact face, that is to say duringthe instances of finger contact, the graphical objects remain visible tothe user. In this context there may be provision that those graphicalobjects for which an instance of finger contact of the assigned fingerhas been detected are not graphically marked.

With respect to the representation of the selectable operating functionby a respective graphical object, one development provides that eachselected operating function is activated, that is to say, for example, arespective graphical user interface is started or displayed, and in thiscontext the display face is operated in a sub-divided fashion in such away that a respective dedicated sub-region of the display face isavailable for outputting respective functional data simultaneously foreach activated operating function. In particular, a plurality ofoperating functions are activated. Therefore, for example the displayface is sub-divided in such a way that each operating function candisplay its graphical user interface. Thus, a user can therefore make apersonal selection of operating functions and simultaneously has all theoperating functions available on the display face, that is to say theycan be operated by him.

In the text which follows, developments are described which relate tothe detection of the finger contact and the identification of thefingers.

According to one development, the finger contact of the at least onefinger on the contact face is detected by a sensor matrix withproximity-sensitive and/or contact-sensitive sensors, that is to say,for example, capacitive sensors. As a result, an instance of fingercontact can be detected, for example, on a touchscreen or a touchpad.Additionally or alternatively there can be provision for this purposethat the finger contact of the at least one finger on the contact faceis detected by a camera. For this purpose, an object recognition devicecan evaluate camera images or image data. The camera can be, inparticular, a time-of-flight camera, (TOF camera) by which distance datacan also be determined, which data can be used during the detection of adistance of a finger from the contact face.

One development provides that in order to identify the at least onefinger which contacts the contact face an arrangement of contact pointsof the contacting fingers on the contact face is used as the basis. Itcan therefore be provided, for example, that the graphical objects onthe display face are displayed in a predetermined arrangement pattern,and the contact points of the fingers are checked to determine whichcontact pattern they form. By comparing the arrangement pattern with thecontact pattern it is possible to specify which finger contacts thecontact face. Additionally or alternatively to the analysis of thearrangement of contact points, one development provides that the fingersof the hand are detected by the time-of-flight camera, and each fingerof the hand is recognized in 3D image data of the time-of-flight cameraand a respective distance of each finger from the contact face isdetermined. As a result it is then known which finger contacts thecontact face. The virtual contact face can also be implemented this way.

In order to carry out the method, an operating device is provided fordetecting a user selection of one or more or all of the operatingfunctions from a predetermined set of a plurality of operating functionsof the operating device. The operating device includes a detectiondevice for detecting an instance of finger contact of at least onefinger of a hand of a user (in particular a plurality of fingers) on apredetermined contact face of the operating device and for determiningwhich of the fingers of the hand contacts the contact face. Furthermore,the operating device includes a control device which is configured toassign one finger of the hand of the user to each of the operatingfunctions of the set and to specify each of the operating functionswhose assigned finger contacts the contact face as the user selection asa whole, and to signal the user selection to a subsequent process of theoperating device by selection data which identify each selectedoperating function. The control device can be implemented, for example,on the basis of a processor device, for example a microcontroller ormicroprocessor. As a whole, the operating device is configured toimplement an embodiment of the method. For this purpose it is, forexample, also possible to provide a corresponding program module whichcan be executed, for example by the processor device of the controldevice and which, during the execution process, implements or carriesout the method executed by the control device.

As already stated, the operating device can, in particular, also be usedto advantage in a motor vehicle. The motor vehicle may be configured asa car, in particular as a passenger car or as a truck.

BRIEF DESCRIPTION OF THE DRAWINGS

In the text which follows, an exemplary embodiment is described thatmakes the aspects and advantages more apparent and more readilyappreciated, taken in conjunction with the accompanying drawings ofwhich:

FIG. 1 is a schematic illustration providing a block diagram and frontview of an embodiment of the operating device during the display ofgraphical objects which each represent a selectable operating functionof the operating device;

FIG. 2 is a schematic illustration providing a block diagram and frontview of the operating device in FIG. 1 during the detection of aplurality of instances of finger contact on a contact face;

FIG. 3 is a schematic illustration providing a block diagram and frontview of the operating device during the detection of a confirmationgesture;

FIG. 4 is a schematic perspective view of a hand of a user during aselection of two other operating functions of the operating device; and

FIG. 5 is a schematic illustration of a display face of the operatingdevice, which can be operated in a sub-divided fashion in such a waythat a plurality of operating functions can output respective functionaldata simultaneously.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

An exemplary embodiment is described in the text which follows. In theexemplary embodiment, the described components of the embodiment eachrepresent individual features which can be considered independently ofone another and which each also may be used independently of one anotherand can therefore also be considered individually or in anothercombination than that shown. Furthermore, other features than thosealready described can also be added to the described embodiment.

In the figures, functionally identical elements are respectivelyprovided with the same reference symbols.

FIG. 1 shows an operating device 1 which can be made available, forexample, in a motor vehicle 2 or (not illustrated) in a portable mobileterminal, for example a smartphone or a tablet PC. The operating device1 can have a display device 3 with a display face 4. The display device3 can be, for example, a screen or a head-up display. A display contenton the display face 4 can be controlled by a control device 5 which canbe implemented, for example, on the basis of a microcontroller or amicroprocessor. The control device 5 can control, for example, devicecomponents 6, for example vehicle components of the motor vehicle 2, bya corresponding control signal 7. The device components 6 areillustrated in FIG. 1 merely by a single element. For each of the devicecomponents 6, the control device 5 can make available a respectiveoperating function 8 by which, for example, a graphical user interfacefor operating the respective device component or a control function fortriggering a device function of the respective device component can beimplemented. The control device 5 can represent the operating functions8 by corresponding graphical data 9 on the display face 4 of the displaydevice 3, respectively by a graphical object 10, for example an icon orlettering, or a menu entry.

The operating device 1 makes it possible for a user now to select aplurality of the graphical objects 10 simultaneously and as a resultmake a selection, for example in a menu, of the operating functions 8which are to be offered to the user on the display face 4, that is tosay a selection of only the graphical objects 10 which are still to bedisplayed. It is also possible to implement simultaneous activation of aplurality of the operating functions 8 by a corresponding simultaneousselection of a plurality of the graphical objects 10. In order torecognize or detect the user selection, the operating device 1 can havea detection device 11 which can include, for example, a camera 12, inparticular a TOF camera, and/or a proximity-sensitive and/orcontact-sensitive sensor matrix 13. The sensor matrix 13 can be madeavailable, for example, on the display face 4, that is to say in thiscase the display device 3 is a touchscreen. A detection range 14 of thecamera 12 can also be directed toward the display face 4.

FIG. 2 illustrates how the control device 5 detects that the usercontacts the contact face 4 with fingers 15 of his hand 16. The contactface 4 constitutes a contact face in this case. The control device 5,assigns the fingers 15 of the hand 16 respectively to the operatingfunctions 8. Accordingly, one of the fingers is also respectivelyassigned to each of the graphical objects 10. A possible assignment isillustrated in FIG. 2 by corresponding letters A, B, C, D, E.

The detection device 11 is able to detect which finger 15 the user usesto contact the contact face in the form of the display face 4. In theexample shown, these are the fingers A, B, that is to say the thumb andthe index finger, for example.

Corresponding instances of finger contact 18 can be detected, forexample, on the basis of 3D image data 12′ of the camera 12 and/orcontact data 17 of the sensor matrix 13 by the detection device 11 andsignaled to the control device 15. It is also signaled which of thefingers 15, that is to say the fingers A, B here, carry out theinstances of finger contact 18 on the contact face in the form of thedisplay face 4.

FIG. 3 illustrates how the user confirms, by a confirmation gesture 19,the selection of the operating functions 8 selected by the instances offinger contact 18. The control device 5 deletes the other, non-selectedgraphical objects 10 on the display face 4 by adapting the correspondinggraphical data 9. It is then possible to provide that only an operatingmenu which has been adapted in this way is then displayed to the user infuture on the display face 4.

The confirmation gesture can be, for example, a movement of the handtogether with the fingers 15 along a predetermined movement direction,for example upward in FIG. 3. When the confirmation gesture 19 isdetected, selection data 20 which describe which of the operatingfunctions 8 the user has selected are generated by the control device 5.

FIG. 4 illustrates the finger position 21 with which the user couldselect two different operating functions 8 on the operating face, namelythose operating functions 8 which are assigned to the fingers A, E.

FIG. 5 illustrates, as an alternative to the configuration of anoperating menu, how the selected operating functions 8, as described bythe selection data 20, can be activated simultaneously and how asub-face or a sub-region 22 is made available for each operatingfunction on the display face 4 in accordance with the number of selectedoperating functions, in which sub-region 22 each activated operatingfunction 8 respectively outputs functional data, for example displaysstatus data or makes available contact faces or operating faces forreceiving user inputs for the respective operating function 8. FIG. 5shows by way of example how four operating functions 8 can be displayedby the display device 3. For example, an air conditioning setting 23, anavigation operating device 24, a media playback operating device 25 forplaying back, for example, music files S1, S2, and a seat-settingfunction 26 for setting a sitting position can be represented as arespective operating function 8 simultaneously on the display face 4 ina respective sub-region 22.

Therefore, the operating device 1 permits a multi-contact gestureoperating system or multi-touch gesture operating system for menuselection of a plurality of operating functions 8. The gesture operatingsystem on the contact-sensitive display face 4 of a touchscreen 3 orelse on a virtual contact level or a contact level of a touchpad servesto detect a desired arrangement of the menu structure or forsimultaneously activating a plurality of operating functions.

In the case of the menu arrangement it is possible, as illustrated inFIG. 1, initially to display a standard orientation of the individualmenu points in the form of a respective graphical object 10, for examplefor radio or navigation. FIG. 2 illustrates how the selection is made bydetection of instances of finger contact 18 or finger taps, wherein thenumber and the finger position are determined. FIG. 3 shows how all themenu items in the form of the graphical objects 10 apart from theselected ones are removed by sliding upward or by an alternativedirection of movement.

The described gestures can be detected on the touchscreen by the sensormatrix 13 or can be freely detected in the air by contactlessrecognition by a TOF camera. Corresponding program modules can beadapted with little expenditure to the respective configuration of theoperating device 1.

Overall, the example shows how a multi-touch gesture operating systemprovides for menu selection.

A description has been provided with particular reference to preferredembodiments thereof and examples, but it will be understood thatvariations and modifications can be effected within the spirit and scopeof the claims which may include the phrase “at least one of A, B and C”as an alternative expression that means one or more of A, B and C may beused, contrary to the holding in Superguide v. DIRECTV, 358 F3d 870, 69USPQ2d 1865 (Fed. Cir. 2004).

1-10. (canceled)
 11. A method for detecting a user selection of one ormore or of all of a predetermined set of operating functions of anoperating device, comprising: displaying all of the operating functionsof the predetermined set, each represented by a dedicated graphicalobject, on a display face; assigning fingers of a hand of a user to eachof the operating functions of the predetermined set, by a controldevice; detecting an instance of finger contact of at least one of thefingers of the hand, by a detection device, on a predetermined contactface of the operating device; determining each selection finger, amongthe fingers of the hand, that contacts the predetermined contact face;specifying a whole user selection, including each selected operatingfunction corresponding to each selection finger, by the control device;signaling the whole user selection to a subsequent process of theoperating device by selection data identifying each selected operatingfunction; displaying each selected operating function respectivelyrepresented by the dedicated graphical object on the display face in thesubsequent process while excluding the dedicated graphical object of anyoperating function excluded from the whole user selection, the dedicatedgraphical object of each selected operating function being displayed ina first spatial region different than a second spatial region in whichthe finger contact of the at least one of the fingers is detected;activating, in the subsequent process, each selected operating function;and outputting on the display face, in a sub-divided fashion, in arespective dedicated sub-region of the display face, respectivefunctional data simultaneously for each activated operating function.12. The method as claimed in claim 11, further comprising detecting aconfirmation gesture of the hand by the detection device, and whereinthe specifying of the whole user selection is performed only when theconfirmation gesture of the hand is detected by the detection device.13. The method as claimed in claim 12, wherein an overall movement ofthe hand together with the fingers which contact the contact face in apredetermined direction along the contact face is detected as theconfirmation gesture 19).
 14. The method as claimed in claim 13, whereinthe detection device detecting the finger contact of the at least one ofthe fingers on the contact face is at least one of a sensor matrix withproximity sensitivity, contact-sensitive sensors and a camera.
 15. Themethod as claimed in claim 14, wherein the determining of each selectionfinger that contacts the contact face includes at least one of detectingan arrangement of contact points of contacting fingers on the contactface, and a sequence of detecting the fingers of the hand by atime-of-flight camera; recognizing each finger of the hand in 3D imagedata of the time-of-flight camera; and determining a respective distanceof each finger from the contact face.
 16. The method as claimed in claim13, wherein the determining of each selection finger that contacts thecontact face includes at least one of detecting an arrangement ofcontact points of contacting fingers on the contact face, and a sequenceof detecting the fingers of the hand by a time-of-flight camera;recognizing each finger of the hand in 3D image data of thetime-of-flight camera; and determining a respective distance of eachfinger from the contact face.
 17. The method as claimed in claim 12,wherein the detection device detecting the finger contact of the atleast one of the fingers on the contact face is at least one of a sensormatrix with proximity sensitivity, contact-sensitive sensors and acamera.
 18. The method as claimed in claim 12, wherein the determiningof each selection finger that contacts the contact face includes atleast one of detecting an arrangement of contact points of contactingfingers on the contact face, and a sequence of detecting the fingers ofthe hand by a time-of-flight camera; recognizing each finger of the handin 3D image data of the time-of-flight camera; and determining arespective distance of each finger from the contact face.
 19. The methodas claimed in claim 11, wherein the detection device detecting thefinger contact of the at least one of the fingers on the contact face isat least one of a sensor matrix with proximity sensitivity,contact-sensitive sensors and a camera.
 20. The method as claimed inclaim 11, wherein the determining of each selection finger that contactsthe contact face includes at least one of detecting an arrangement ofcontact points of contacting fingers on the contact face, and a sequenceof detecting the fingers of the hand by a time-of-flight camera;recognizing each finger of the hand in 3D image data of thetime-of-flight camera; and determining a respective distance of eachfinger from the contact face.
 21. An operating device for detecting awhole user selection of one or more or of all of a predetermined set ofoperating functions of the operating device, comprising: a display faceinitially displaying all of the operating functions of the predeterminedset, each operating function represented by a dedicated graphicalobject; a contact face; a detection device detecting an instance offinger contact of at least one finger of a hand of a user on the contactface of the operating device and determining each selection finger,among the fingers of the hand, that contacts the contact face; and acontrol device configured to assign the fingers of the hand to each ofthe operating functions of the predetermined set, specify the whole userselection, including each selected operating function corresponding toeach selection finger, signal the whole user selection to a subsequentprocess of the operating device by selection data identifying eachselected operating function, display each selected operating functionrespectively represented by the dedicated graphical object on thedisplay face in the subsequent process while excluding the dedicatedgraphical object of any operating function excluded from the whole userselection, the dedicated graphical object of each selected operatingfunction being displayed in a first spatial region different than asecond spatial region in which the finger contact of the at least one ofthe fingers is detected, activate, in the subsequent process, eachselected operating function, and output on the display face in arespective dedicated sub-region of the display face, respectivefunctional data simultaneously for each activated operating function.22. A motor vehicle, comprising: a chassis; and an operating device,including a display face initially displaying all of the operatingfunctions of the predetermined set, each operating function representedby a dedicated graphical object; a contact face; a detection devicedetecting an instance of finger contact of at least one finger of a handof a user on the contact face of the operating device and determiningeach selection finger, among the fingers of the hand, that contacts thecontact face; and a control device configured to assign the fingers ofthe hand to each of the operating functions of the predetermined set,specify the whole user selection, including each selected operatingfunction corresponding to each selection finger, signal the whole userselection to a subsequent process of the operating device by selectiondata identifying each selected operating function, display each selectedoperating function respectively represented by the dedicated graphicalobject on the display face in the subsequent process while excluding thededicated graphical object of any operating function excluded from thewhole user selection, the dedicated graphical object of each selectedoperating function being displayed in a first spatial region differentthan a second spatial region in which the finger contact of the at leastone of the fingers is detected, activate, in the subsequent process,each selected operating function, and output on the display face in arespective dedicated sub-region of the display face, respectivefunctional data simultaneously for each activated operating function.